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Introduction 



The availability of federal funding for the development and 
implementation of different and sometimes unique instructional 
programs has appeared to be somewhat of a mixed blessing. Without 
such resource Incentive, it is possible, if not probable, that the 
emphasis on structured planning and evaluation of' such programs 
would not have been present, A comparison of a school district's 
local expenditure for planning and evaluation activities versus 
the expenditure of federal monies for evaluation or formalized 
planning activities will most likely support my contention. Given 
l!>he vast amounts of time and the related expense involved in the 
myriad of activities necessary to develop a sound plan, and then 
the paperwork and time needed to effectively monitor the implemen- 
tation of the plan, it is no wonder that local school* administra-- 
tors are asking for some relief from the time consuming planning 
tasks. Unfortunately, the shrinking resources and the increased 
inflation; demand more and better plannin;-; rather than less. Tools 
are needed to aid local school decisior makers in their delibera- 
tion. 

This paper will review some of the shortcomings identified with 
the formalized planning and evaluation systems presently in existence. 
It will also review some of the forgotten assumptions supporting 
the development of these systems. It will review some of the pro- 
posed models for the evaluation- of programs in terms of costs. Pin- 
ally It will propose two models which because of their simplicity 
could be of benefit to the educational decision maker who does not 
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have economists operations researchers, and the like on his plan- 
ning sta^ff . 
Evaluation 

It is of no surprise, to be told that many pro£,rams lack clear 
objectives upon which to base an evaluation design. Provus (1971) 
as well as most authors on this paper's reference list all iden- 
tify the lack of clear program objectives, and clear descriptions" 
of program processes as hampering the evaluation of the program. 
Guba (1969) elaborates on the shortcomings ' of elaboration and cites 
the following "lacks 

1. Lack of adequate definition of evaluation. 

2. Lack of adequate evaluation theoi^y, 

3* Lack of knowledge about decision processes. 
^. Lack of criteria. 

5. Lack of approaches differentiated by levels. 

6. Lack of mechanisms for organizing, processing and 
reporting evaluative information. 

7. Lack of trained personnel.' 

Walberg (1970) cites the .influence of Ralph Tyler's writings 
on educational evaluation, the controversy on objectives, the pro- 
blem with programming, the inconsistency of educational research 
to identify learning experiences whicl work for all students, the 
lack of generalizability of evaluation, and the need for a more 
useful'; evaluation. 

New books by Provus (1971) the Phi Delta Kappa National Study 
Commission (19T1) and others, are attempts to eliminate these eval- 
uation shortcomings previously cited. 
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Decision making 

The Phi Delta Kappan National Commission (1971) defined eval- 
uation as the ^'process of delineating, obtalninf;:; and nrovldlnn; use- 
ful Information for identlfylnf^ decision alternatives." One of 
the difficulties with providing; decision alternatives for decision 
makers, is that the decision making settinr;s are frequently too 
poorly defined. In this era of emphasis on manap^ement information 
systems J, this can result in a decision maker receiving roams of 
computer generated data but precious little decision maklnr. infor- 
mation. V/ard and Lehman state (1972) "The point is that decision 
levels 'must be served with infor^mation that" applies to each level, 
but previous to now a suitable process f or . des cribing decision set- 
tings has not been formulated." 

Eight years earlier, Dill^s (1964) chaoter in the 1963 NSSE 

Yearbook cites the following quotations: 

The task of "deciding" pervades the entire administrative 
organization quite as much as the task of "doing". The 
central function of administration is directing and control- 
ling the decision-making process. The executive is a decider 
and not a doer. 

Dill's entire chapter emphasized the Increased attention given by 
specialists in administration to the art or science of decision 
making. He indicated that most research revolved around finding 
answers to three questions: How are decisions made? V/ho should' 
make decisions? How can we make better decisions? It is the last 
question--How can better decisions be mad — that the major part 
of the paper will be addressed. 



Kershaw and McKean (1959) after studyinr, the ootential for 

using systems analysis techniques in education summari^^ed: 

After having studied the educational process, the pro- 
blems of measurement, and the data that are becoming 
available, we conclude that it v/ill soon be feasible to 
make comparisons of this sort that can help administra- 
tors and others choose iTnoroved educational systems. It 
will be necessary first, hov/ever, for more work to be 
done toward estimatinp; the '^input-output relationships" 
in education. 

Prom early observations such as this and the expanded -use of PPBS 
in the government and defense Industries, come similar prooonents 
in education. Hartley (1968) is the more common proponent but 
Banghart (1969) and Van Dusseldorp (1971) also encourage its adop- 
tion through the umbrella term of system.s analysis. Alioto and 
Jungherr (1969) indicate the reason for their district p;oing to 
PPBS in their article entitled '^Usinp: PPBS to Overcome Taxpayers 
Resistance . '' Specific advantaf^cs to the particular PPBS proposed 
by their district or committee can be found in articles by the 
Trenton Public Schools (1971), V/estern New York Study Council (1969) 
and National Committee on Educational Finance (1968) to name a few. 
■A more complete review can be had through the use of the AASA (1970) 
Collection of Abstracts entitled "A Collection of ERIC Documents 
Resumes on Program Budgeting and Cost Analysis.^' An article by 
Bibbs, Rath and Kent provides some lnsip;hts p;ained by the Skokie 
School district after one year's experience with P.P.B.S. . 

Lt is the introduction and use of P.P.B.S. which has brought 
the terms cost effectiveness, cost benefit analysis, cost-util- 
ity, utility-cost ratios to the forefront of educational pubii- 
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cations • In fact. It Is the- ability ( ? planninc systems such as 
PPBS and its structured attempts to link costs v/ith programs, and 
programs with specified desired cutouts that makes PPBS so attrac- 
tive to school administrators. V/ith the current demand for a visi- 
ble accounting of educational outputs for the budp;ctary inputs, 
planning systems which automatically provide information about pro- 
gram outputs are regularly beinc sought. Unfortunately, PPBS, and 
its associated cost effectiveness analysis or cost benefit analy- 
sis , vjill not serve as a miracle elixir to cure poor decision mak- 
ing. If used for unintended purposes or in unintended ways^ it may 
heir to make poorer decisions. 

Kaufmann (1968) identifies some serious misconceptions about 
cost-benefit analysis. Some of these shortcomings are cited below 
. with a short comment by this author for clarification: 

1. ^'It is merely a subterfuge for seeking; to conduct educa- 
tion on a least-cost basis. ^' Proponents of cost-effect- 
iveness analysis usually seek efficiency of operations 

so that the savings could be applied to other educational 
needs for which funds are lacking. 

2. "Some things can't be measured." flaybe.so, but if the 
benefits of the program cannot be described in some sub- 
stantive way, how will people decide if they wish to spend 
more money to support it. 

3* "The cost benefit analysis ignores the political fac- 
tors./' This isn't true. In fact. It can Help to iden- 
tify the costs of making so-called political decisions. 



Purno (1969) agrees v;ith the potential of PPBS and the associated 
cost . analysis , but cites many of the forgotten short comlnp:s , the 
largest of which is the data collection, organization, and util- 
ization. He cites the need for a very sound position control sys- 
tem to support the financial information system, and remarks about 
the lack of such systems in the major cities where the largest 
benefits of such cost analysis could be accrued. 
Cost-Effectiveness Analysis 

Much of the difficulty surrounding the use of cost effective- 
ness analysis lies with the lack of clear definitions as well as 
satisfactory methods of measuring; costs and benefits. In the re- 
view of literature cited by the Western Nev/ York School Study Coun- 
cil (1969) the only stated difference between cost-benefit and 
cost-effectiveness analysis, was the choice of output under scru- 
tiny. If the program's benefits could be converted to dollar terms, 
it was considered to be cost-benefit analysis. If the benefits 
could not be so converted^ the analysis was considered to be termed 
cost effectiveness. 

The workshop reported by McAbee (1968) offered the following 
definitions: 

Cost Effectiveness is the cost per program unit produced. 
Cost Benefit is the relationship between the cost of the pro- 
duct and value of the benefit of producing it. 

Davie (1965) identified some other limitations in the use of bene- 

fit-cost analysis. One was the dilemma of considering a program's 

value from only a local viev/point since it may conflict with the 

value of a larger reference group. Another interesting problem 
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Is due to the differing values which people place on money. The 
subtle semantic difference of labeling somothinp; an educational 
'^inves tment " versus an educational '^expenditure" implies this dif- 
ference in attitude tovjard the value of money. V/elty's (1971) pa- 
per describes diiferent types of cost-benefit analysis in terms of 
the v;ay in which the analysis is structur-^d. He describes those 
studies relating historical data on costs and estimated benefits 
via some type of multi-variate analysis as descrlotlve cost-bene- 
fit analycilt:. The other types of studies might- be labeled as struc- 
tural or experimental cost-benefit depending upon the development 
of the production function under consideration. Despite the vari- 
ous identified shortconiingi:. , moro and more articles are being v/rit- 
ten about the advantages of cost benefit analysis and cost effect- 
iveness analysis. A review of cost-benefit — PPBS literature fre- 
quently finds Quade's (1965) article on Cost Effectiveness cited. 
Ills earlier observations are still pertinent. He first identi- 
fies the five elements of analysis which are: the objective; the 
alternatives; the costs; the model; and the criterion for measure- 
ment. His comments on costs are most approprJ :ite for he states: 

In analysis for a future time porjod^ most costs can be 
measured in money, but their true measure is in terms 
of the opportunities they preclude. 

It is this equating of lost opportunities to costs, which intro- 
duces a difficult conceptual problem for the users of cost effect- 
xveness analysis. The models identified in the last portion of 
the paper try to bridge this gap with the use of utility measures. 
Quade's other observations- deal with. the caveats of analysis. 
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They are : 

!• All analysis is necessarily incomplete due to time 

and money constraints. Frequently hindsight can make 
what was originally a good decision appear to be a 
bad decision. 

2. Most measures of effectiveness are approximate and in 
fact the cost measures are frequently estimates also. 

3* No satisfactory way to predict the future exists and 
so any estimate of optimum policy can only be in terms 
of the best guess as to what the future holds. 

4. All analysis of choice falls short of scientific re- 
search. Human jujjgment is used in desicning the analy- 
sis; in deciding what alternatives to consider, what 
factors are relevant , vjhat the interrelations betv/een 
the factors are, and what numerical values to choose; 
and in analyzing and interpreting the results of the 
analysis. This fact that judgment and intuition per- 
meate all analysis should be reiuembered when v;e exam- 
ine the results that come, with apparent high preci- 
sion^ from analysis. 

With Quade^s cautions in mind, an increasing number of individuals 
have been looking at the educational process and trying to develop 
models which are appropriate for evaluating educational programs 
in terms of cost and effectiveness. All of the studies are attempt 
to provide a de.cision maker v/ith better information than is current 
ly held. Temkln (1971) has proposed the use of a tool or unit 
which he calls the Elasticity of Cost Effectiveness. Abt (1966^ 
1969) cites approaches taken in the evaluation of Indian education 
programs, and In the analysis of Title I ESEA projects. Storms- 
dorfer (1972) and Davie (I965) have used cost effectiveness analy- 
sis models for the evaluation of Vocational Education projects. 
Webster (1972) uses cost-ef f ect iyen^ss at its simplest level of 
analysis in her comparison of reading programs in terms of their 



costs to produce 0,1 unit of pain In readinr, achievenent. Cleck- 
nor (1971) offers a critique of five different models and pro- 
poses changes which can eliminate his identified shortcomings . 
Badran (1970);, Harmon (1970) and Mood (I967) offer rroneral com- 
ments on the worth of cost effectiveness analysis and offer ap- 
proaches to be utilized in carrying out such analysis. Chuanp; 
(1972) describes a planning system dpsif^ned to convert broad 
statements of goals into more refined decisions v/.Lthin a budget 
context. Haggart (1971) proposes a variety of suggestions regard-- 
Ing the costing of programs, though her earlier work wi.rh Carnen- 
ter (1970) is an excellent introduction to the use of cost effect- 
iveness analysis in education. It is the work of researchers such 
as these which has continued to provide assistance to educators 
v;ho must make decisions. Decision making is seldom easy 5 and the 
Information to support the decision even more difficult to obtain. 
It is the belief that better information, provided in a planned 
setting, will permjt more rational and hopefully '"fitter, deci- 
sions, that prompts the recommendatio:i of co::: t -1/ il.l ty analysis 
as an aid to decision making. 
Present Model Shortcomings 

The previous pages have provided a cursory review of the lit- 
erature relative to P.P.B.S. 5 decision making, evaluation, and par- 
ticularly to cost effectiveness analysis. V/hile the shortcomings 
of cost effectiveness analysis have been identified, it is hoped 
that the positive potential benefits are still apparent. The 
one drawback to many of the cited models^ is their need for a team 
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of analysts to bo present on r^chool superintendent ' staffs. If 
P.P.B.S. and its associated cost analyslG is to becone more pre- 
valent in school system decision scttinnn^ it will have to become 
soinewhat easier to use by the r.taff already present. Increases in 
administrative staff tend to be diffloult to obta':" 'iurin^, tj::ies 
of financial pinch. The tv/o models about to be described, repre- 
sent what might be described as tools v;!:ich could be used by iriost 
any school adminstrator and his present staff v/hon faced v/itbi choo- 
sing between alternative prori^anis deslr^ned to meet equivalent Roals . 
The models were chosen for their relative ease of comnutation, lack 
of confusinc jarcon, and yet still able to provide valuable rational 
decision making. 
T anner ^s Model 

The first model is one pi»oposed by C. Kenneth Tanner (1971) 
in his text Desip;ns for Educ at ional Plann inr:. The article by Hughes 
and Tanner (1970) provider a case stuay of this model in ude. No 
attempt will be made there to discusr; the model in its entirety , but 
rather to identify its highlights and in turn its. value in decision 
making. 

Tanner's model attempts to incorporate the use of subjective 
judgments about a prosram's worth. He suggests a :3ayerlan approach 
which will utilize all of the informacion known about the program 
to yield a' series of values which are probabilities of the program's 
future worth and can also serve as a baseline for measuring the 
program's worth at a later time. Tanner (1971) very carefully lists 
all of the assumptions he has made regarding :;he use of the model. 



Some of them~such as, the program elt^ments are ap;reeable to the 
formulation of clear, cone*' ^o^-^urable objectives and the unit 
cost of each program e" determined — are the same 

kinds of assumptions fou. ^ any model of this type. S.r,..' as- 

sumntions do not seem to limit the value of the tool. 

Tanner*?^ model is described in more detail in his text, but 
the following steps should provide some idea of its formulation 
and use. When following along, one should assume -that the plan- 
ner using the tool wishes to assess the future effectiveness of 
a program consisting of ten courses, and in particular, iden- 
tifylng the most effect course of the ten. In addition, assume 
that the students* success in each course Is reflected by the num- 
ber of objectives completed in that course, and that these success 
scores have been collected by the planner. 

Step- One Determine the percent of behavioral objectives achieved 
for one program clement or course. 

Step Two Determine what the value judgments, regarding the gen- 
eral program objective, were per program element. These 
Judgments might come from the participants, the teach- 
ers/ some panel of experts, etc. and the judgments are 
to be captured via some attltudinal Instrument. 
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Table 1 

Future Program Utilities Based on the Initial Sample 



Program 
Element 
(Course) 



1 

2 
3 

5 
6 

7 
8 

9 

10 



Prior 
Distribution 
(Values from 
Step 2) 

.115 

.0l|,8 
.073 
.091 
.109 
.127 
.152 
.Di|8 
.16^1 



Sample 
Distribution 
(Sten 1) 



.89'^ 
.916 
.832 
.896 
.980 
.935 
.769 
.799 
.7^19 
.950 



Joint 
Probabilities 
(Step 1 X 

Step ?) 

.036 
.133 
.040 

.065 
.089 

.102 
.098 
.121 
.036 
.156 



Posterior 
Probabilitie 
JPl/ JP-L 

.0^11 
.152 

.074 

.102 
.116 
.112 
.138 
.Oi|l 
.178 
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The purpose pf using this Bayesian approach is to determine 
the utility (worth) of a given program element v/ithin the group 
of ten elements • The Bayesian posterior probability distribution 
serves as an estimate of future program worth or utility- It is 
important to note, that utility value was achieved by combin-- 

ing attitudinal judgmer ^' - X each program's element (Column 1) 
with observed quantitative data about the program's worth (Column 2). 
Step Four Integrate the program element costs with the calculated 
utilities. This is done by calculating the condition a l worth per 
program element which is defined as the abs olute difference be- 
tween the standardized cost of that program element and^ the selling 
prices of tiie other elements. Naturally the program costs will 
have to be standardized in some manner as to permit valid compari- 
sons. Since public education is not expected to operate at a pro- 
fit, it is assumed that selling price and cost are equal on any 
given program. Table 2 (Tanner, 1971) shows the conditional worth 
calculated for program element 1, The same calculations would be 
performed for all 10 elements. 
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Table 2 

Conditional V/orth of Program Element 1 



Program 


Selling 


Cost of 1 


Conditional 


Element 


Price 




■ VJorth 


1 


18 


18 


. 0 


2 




18 


3 


3 


■'■ 


18 


0 


k 


17 


18 


1 


5 


23 


18 


5 


6 


in 


18 


k 


7 


26 


18 


8 


8 


19 


18 


1 


9 


22 


18 


k 


10 


23 


18 


5 


Step Five 


Calculate the 


expected opportunity 


loss, which 



sum of the products of the utilities and the conditional worth per 
column per program element. Table 3 represents the calculations 
necessary to find the expected opportunity loss related to the 
choice of program alternative 1. 
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Table 3 

Expected Opportunity Loss for Program Element I 

Progrc/,' - Utilities Conditional Expected 

Element Worth of Ooportunity 

Element I " Loss 

(dollars) (utiles) 



1 


.on 


0 




0 


2 




3 




.456 


3 




0 




0 


^ 




1 




074 


5 


.102 


5 




.510 


6 


.116 


H 




.464 


7 


.112 


8 




.896 


8 


.138 


1 




.138 


9 


.Oill 


H 




.164 


10 


.178 • 


5 


Total 


.890 
3.592 



Table 4 r. fleets the total loss for the entire nrogram. 

Table 4 

Expected Loss for Total Program 

Program Expected 

Element Loss (utiles) 

1 3.592 

2 2.806 

3 3.592 
^ 4.212 

5 . 3.156 

6 , 6.516 ■ 

7 5.484 

8 3.146 

9 2.980 . 
iliO 3. '156 
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One should note that the value of the Expected Loss repre- 
sents a combination of the utility of the program and the rela- 
tive costs of the program as reflected in its conditional worth. 
In addition^ it should be evident that in most cases, the opti- 
n?al program choice is that v;hich provides the leasb expected 
loss» 

This very brief explanation in no way does full justice to 
the Bayesian model proposed by Tanner, -It is presented only to 
provide a glimpse as to its ease of use and genuine worth in 
terms of its ability to aid decision making. 
Milwaukee Mode l 

The second model being proposed 'is similar to the first in 
its attempt to take advantap;e of orderly opinion about program 
worth. This model is explained more thoroughly in Costa and Gi- 
roux (1971) and the Association of School Business Officials' 
final report. As in the previous example , the main purpose is 
to give a general idea as to the model's workings and to high- 
light its ability to aid decision making. The r.icdel assumes that 
enough is known about the project such trat measureable objec- 
tives can be created which will serve as indicators of the pro- 
ject's purpose. Furthermore, the activities within the program 
can be categorized into unique processes. Frequently, these pro- 
cesses will be input oriented, such as the use of a teacher's 
aide as a tutor, or the use of a particular set of instructional 
materials. The model is designed to assist the decision maker . 
in identifying the most important objectives of the program, the 
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most important processes in the program, and the perceived rela- 
tionships between the two. In addition, it also provides for the 
introduction of standardized cost data to develop utility/cost 
ratios for the purpose of identifying the economic and value con- 
sequences of various process choices. This approach r ] ' 
most beneficial in situations where whole program are not about 
to be discarded, but rather selective modification will be used 
prior to. refunding of the program., 

The steps involved in utilizing such a model is as follows: 
Step 1 In some structured way, select a jury or panel of experts 
capable of making Informed judgments about the program. In some 
cases, this jury may include program participants themselves. 
Step 2 Each jury member is then asked to rank each of the objec 
tlves in terms of their importance. Importance Is defined as the 
degree to which the jury would like to see the objective achieved 
at the termination of the program. 

Step 3 Calculate the mean rank for each objective by summing 
the ranks and dividing by the number of rankers for each and ev- 
ery objective. Should there be great variance of opinion on any 
objective, the planner may wish to use some consensus forming 
strategy such as the Delphi method to reduce the variance. 
Step 4 The objectives are then rank ordered in terms of their 
mean rank. In addition, each objective Is given a weighted 
value of importance equal to the Inverse of its rank. An objec- 
tive ranked tenth in a group- of ten objectives would be. given a 
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weight of one (1). 

Step 5 Have each jury member rank order the Processes In terms 
of each objective. 

Step 6 Sum the ranks for all Drocr-^'-'^s . octlves. Let 

^ eq.uals the score or rank f^lven by ranker 1 on nrocess 
j for objective k. 

Step 7 The proportion of c:cntrlbutlon that a pro.cess makes to- 
ward the attainment of a particular objective is equal to the to- 
tal score of a process for aa objective, attained in step 6, di- 
vided ky the sum of the total scores for all processes for that 
objective. If there are q rankers , m objectives , and n process- 
es, then the proportion of contribution that process 1 makes to 
the achievement of objective 1, equals ^^Sj^^p^^^ ^^..'^^S(i.^p^O^-?.C, 
Step 8 Calculate the utility value each nrocess. Utility 
is defined as the de^^ree to which a process is perceived to con- 
tribute to the attainment of an objectlvr or objectives. The util- 
ity value of each process Is^ -eonsldored Ir be a function of how 
much that process helps to a^ecomplish each obj'ective and the re- 
lative iTTiportance of the objer/tive. In other words, the process 
with the highest utility valine is that process which contributes 
ttie most to sishieve the most Important objective. 

utility Value of lOOx (proportion of contribution 



Process i. = "izl^ of process i for Objective k 

IL^ I X 

Weight Value of Objective k) 

Usmally the rvalue inside the brackets is multiplied by 100 so 

as to deal itflth whole numbers-. 
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Step 9 The utility/cost ratio for each process Is equal to the 
utility value of the process di'vlded by the standardized cost for 
the process • 

As with the previous model , this short explanation of the 
models' working does provide the complete explanation it de- 
serves. In addition^ this short explanation did not identify the 
limitations Inherent in the model. As stated in the introduc- ■ 
tion however 3 no model is without some limitations, and so what- 
ever benefits these two approaches might offer local school deci- 
sion makers, should be investigated and considered for adoption 
if appropriate. 

Hopefully what is evident in both models, is a structure 
i^hlch allows for rational systematic program analysis and its 
subsequent aid in decision making. 
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